10/03/2024 - 16/03/2024

Made some changes to frontend code to fix some issues:

common/AMC13.cpp:84
    -ss_ip_hardcoded << "192.168." << byte3 << "." << (13 + sel);
  +ss_ip_hardcoded << "192.168." << byte3 << "." << (188 + sel);
common/AMC13.cpp:84
    -ss_ip_hardcoded << "192.168." << byte3 << "." << (13 + sel);
  +ss_ip_hardcoded << "192.168." << byte3 << "." << (188 + sel);

Change the magic number in this function to correspond to our IP. I hate this code the more I read it.

MasterGM2/frontend.cpp:595
  +strcpy(master_settings_odb.encoder_fe, "AMC13000");
MasterGM2/frontend.cpp:595
  +strcpy(master_settings_odb.encoder_fe, "AMC13000");

overwrite this setting that was set to "AMC1300" as opposed to "AMC13000" I'm not sure where it originally gets set. I think the extra 0 is a result of Lawrence's hijinx to make frontends have 3 digits as opposed to 2.

10/03/2024 21:18

Made some changes:

MasterGM2/AMC1300.cpp:127
  } else if (ver_read != ver_odb) {
    std::string error_msg = "/AMC1300/Settings/AMC13/: Conflict: \"T1 Firmware Version Required\" - Read version: " + std::to_string(ver_read) + ", ODB version: " + std::to_string(ver_odb);
    cm_msg(MERROR, __FUNCTION__, error_msg.c_str());
    
    return FE_ERR_HW;
}
MasterGM2/AMC1300.cpp:127
  } else if (ver_read != ver_odb) {
    std::string error_msg = "/AMC1300/Settings/AMC13/: Conflict: \"T1 Firmware Version Required\" - Read version: " + std::to_string(ver_read) + ", ODB version: " + std::to_string(ver_odb);
    cm_msg(MERROR, __FUNCTION__, error_msg.c_str());
    
    return FE_ERR_HW;
}

This gives output on how the versions differ, I got:

[MasterGM2000,ERROR] [AMC1300.cpp:129:AMC13_init,ERROR] /AMC1300/Settings/AMC13/: Conflict: "T1 Firmware Version Required" - Read version: 33063, ODB version: 33087
[MasterGM2000,ERROR] [AMC1300.cpp:129:AMC13_init,ERROR] /AMC1300/Settings/AMC13/: Conflict: "T1 Firmware Version Required" - Read version: 33063, ODB version: 33087

So I changed the ODB parameter Equipment/AMC13000/Settings/AMC13/T1 Firmware Version Required to be 33063 and got past the error. I did similar for the T2:

MasterGM2/AMC1300.cpp:142
else if (ver_read != ver_odb) {
    std::string error_msg = "/AMC1300/Settings/AMC13/: Conflict: \"T2 Firmware Version Required\" - Read version: " + std::to_string(ver_read) + ", ODB version: " + std::to_string(ver_odb);
    cm_msg(MERROR, __FUNCTION__, error_msg.c_str());
    return FE_ERR_HW;
  }
MasterGM2/AMC1300.cpp:142
else if (ver_read != ver_odb) {
    std::string error_msg = "/AMC1300/Settings/AMC13/: Conflict: \"T2 Firmware Version Required\" - Read version: " + std::to_string(ver_read) + ", ODB version: " + std::to_string(ver_odb);
    cm_msg(MERROR, __FUNCTION__, error_msg.c_str());
    return FE_ERR_HW;
  }

Output:

[MasterGM2000,ERROR] [AMC1300.cpp:144:AMC13_init,ERROR] /AMC1300/Settings/AMC13/: Conflict: "T2 Firmware Version Required" - Read version: 45, ODB version: 46
[MasterGM2000,ERROR] [AMC1300.cpp:144:AMC13_init,ERROR] /AMC1300/Settings/AMC13/: Conflict: "T2 Firmware Version Required" - Read version: 45, ODB version: 46

changed Equipment/AMC13000/Settings/AMC13/T2 Firmware Version Required to match 45 instead of 46

10/03/2024 21:38

To get around these errors:

FC7_init(175): FC7 Board Presence Check
[MasterGM2000,ERROR] [AMC1300.cpp:183:FC7_init,ERROR] /AMC1300/Settings/FC7-11/: Board Absent
FC7_init(175): FC7 Board Presence Check
[MasterGM2000,ERROR] [AMC1300.cpp:183:FC7_init,ERROR] /AMC1300/Settings/FC7-11/: Board Absent
FC7_init(174): FC7 Board Presence Check
Unable to send RAW command (channel=0x7 netfn=0x30 lun=0x0 cmd=0x5)
getAddress(129): Caught Exception
Unable to send RAW command (channel=0x7 netfn=0x30 lun=0x0 cmd=0x5)
getAddress(129): Caught Exception
Unable to send RAW command (channel=0x7 netfn=0x30 lun=0x0 cmd=0x5)
getAddress(129): Caught Exception
Unable to send RAW command (channel=0x7 netfn=0x30 lun=0x0 cmd=0x5)
getAddress(129): Caught Exception
Unable to send RAW command (channel=0x7 netfn=0x30 lun=0x0 cmd=0x5)
getAddress(129): Caught Exception
FC7_init(190): Slot 11: Read FC7 IP Address:
[MasterGM2000,ERROR] [AMC1300.cpp:193:FC7_init,ERROR] AMC13: T1 IP Address Read Failure
FC7_init(174): FC7 Board Presence Check
Unable to send RAW command (channel=0x7 netfn=0x30 lun=0x0 cmd=0x5)
getAddress(129): Caught Exception
Unable to send RAW command (channel=0x7 netfn=0x30 lun=0x0 cmd=0x5)
getAddress(129): Caught Exception
Unable to send RAW command (channel=0x7 netfn=0x30 lun=0x0 cmd=0x5)
getAddress(129): Caught Exception
Unable to send RAW command (channel=0x7 netfn=0x30 lun=0x0 cmd=0x5)
getAddress(129): Caught Exception
Unable to send RAW command (channel=0x7 netfn=0x30 lun=0x0 cmd=0x5)
getAddress(129): Caught Exception
FC7_init(190): Slot 11: Read FC7 IP Address:
[MasterGM2000,ERROR] [AMC1300.cpp:193:FC7_init,ERROR] AMC13: T1 IP Address Read Failure

Had to add the -m 0x20 flag to two ipmitool calls:

common/FC7.cpp:161
  ss_cmd << "ipmitool -I lan -H " << ip << " -U shelf -P shelf -m 0x20 -B 0 -T 0x82 -b 7 -t " << (0x70 + (2 * slot)) << " fru print 0x0 | grep 'FC7' &>/dev/null";
common/FC7.cpp:161
  ss_cmd << "ipmitool -I lan -H " << ip << " -U shelf -P shelf -m 0x20 -B 0 -T 0x82 -b 7 -t " << (0x70 + (2 * slot)) << " fru print 0x0 | grep 'FC7' &>/dev/null";
common/FC7.cpp:87
  ss_cmd << "ipmitool -I lan -H " << ip << " -U shelf -P shelf -m 0x20 -B 0 -T 0x82 -b 7 -t " << (0x70 + (2 * slot)) << " raw 0x30 0x5";
common/FC7.cpp:87
  ss_cmd << "ipmitool -I lan -H " << ip << " -U shelf -P shelf -m 0x20 -B 0 -T 0x82 -b 7 -t " << (0x70 + (2 * slot)) << " raw 0x30 0x5";

This worked, though sometimes the equipment "loses it's mind" as Lawrence put it. I had to wait a minute then run the frontend again after it failed a couple times.

10/03/2024 21:49

I have this error:

[MasterGM2000,ERROR] [AMC1300.cpp:204:FC7_init,ERROR] /AMC1300/Settings/FC7-11/: Invalid "Address Table Location"
       uhal exception: "Invalid node ID 'BOARD.TYPE' specified (contains dots)"
[MasterGM2000,ERROR] [frontend.cpp:903:frontend_init,ERROR] FC7 Initialization Failed
[MasterGM2000,ERROR] [AMC1300.cpp:204:FC7_init,ERROR] /AMC1300/Settings/FC7-11/: Invalid "Address Table Location"
       uhal exception: "Invalid node ID 'BOARD.TYPE' specified (contains dots)"
[MasterGM2000,ERROR] [frontend.cpp:903:frontend_init,ERROR] FC7 Initialization Failed

which I resolved by changing the address table to the "correct" version FC7_CCC.xml in the ODB. I.e. I changed:
Equipment/AMC13000/Settings/FC7-11/Common from
$GM2DAQ_DIR/address_tables/FC7.xml' to $GM2DAQ_DIR/address_tables/FC7_CCC.xml'

11/03/2024 15:51

I added the jumpers to the FMC. I attempted to do it in a way that was consistent with what Lawrence said in an email:

THe firmware is set up to use the “D bank” channels 4-7 for input. As per the instructions, you will need to toggle the micro switch for those for channels for input. For incoming signals that will expect 50 Ohm termination, you’ll want to install the little jumpers I sent along. For the master 40 MHz clock, which will go in on D7, I have seen both synthesizers that expect high impedance but also that expect 50 Ohm termination.

The connections:

I put jumpers on channels D4, D5, D6, and D7 because those are described as the input channels.

11/03/2024 16:44

I became impatient and decided to haphazardly assemble the FMC module on the FC7. I used the stands and screws from the other FMC module we took out, even though they don't really fit and we don't have enough. See images below:

11/03/2024 16:56

When the crate is power cycled, the IPs we set for the two AMC13 FPGAs get reset as well, causing this errror when trying to run the master frontend:

AMC13_init(86): Read AMC13 T2 IP Address: 192.168.1.188
ipbusudp-2.0://192.168.1.189:50001 , file://$GM2DAQ_DIR/address_tables/AMC13XG_T1.xml
AMC13_init(120): AMC13 T1 FPGA Firmware Version Check
Caught Exception
getRegister(207): uHAL Exception: Timeout (1000 milliseconds) occurred for UDP receive from target with URI: ipbusudp-2.0://192.168.1.189:50001

Caught Exception
getRegister(207): uHAL Exception: Timeout (1000 milliseconds) occurred for UDP receive from target with URI: ipbusudp-2.0://192.168.1.189:50001

[MasterGM2000,ERROR] [AMC1300.cpp:124:AMC13_init,ERROR] /AMC1300/Settings/AMC13/: Ethernet Communication Failure
[MasterGM2000,ERROR] [frontend.cpp:895:frontend_init,ERROR] AMC13 Initialization Failed

AMC13_init(86): Read AMC13 T2 IP Address: 192.168.1.188
ipbusudp-2.0://192.168.1.189:50001 , file://$GM2DAQ_DIR/address_tables/AMC13XG_T1.xml
AMC13_init(120): AMC13 T1 FPGA Firmware Version Check
Caught Exception
getRegister(207): uHAL Exception: Timeout (1000 milliseconds) occurred for UDP receive from target with URI: ipbusudp-2.0://192.168.1.189:50001

Caught Exception
getRegister(207): uHAL Exception: Timeout (1000 milliseconds) occurred for UDP receive from target with URI: ipbusudp-2.0://192.168.1.189:50001

[MasterGM2000,ERROR] [AMC1300.cpp:124:AMC13_init,ERROR] /AMC1300/Settings/AMC13/: Ethernet Communication Failure
[MasterGM2000,ERROR] [frontend.cpp:895:frontend_init,ERROR] AMC13 Initialization Failed

I found you could ping the two FPGAs (T1 and T2) with their "default" addresses:

ping 192.168.26.10
ping 192.168.26.11
ping 192.168.26.10
ping 192.168.26.11

To set them back to the "normal" addresses:

cd /home/installation_testing/packages/experiment/lxedaq/amc13/amc13_v1_2_18/dev_tools/amc13Config
./applyConfig.py -n 33
cd /home/installation_testing/packages/experiment/lxedaq/amc13/amc13_v1_2_18/dev_tools/amc13Config
./applyConfig.py -n 33

Then I could once again ping at the "normal" addresses:

ping 192.168.1.188
ping 192.168.1.189
ping 192.168.1.188
ping 192.168.1.189

11/03/2024 17:02

After adding the FMC module as detailed above, I still get these errors:

The value of code is: 0
[MasterGM2000,ERROR] [AMC1300.cpp:331:FC7_init,ERROR] /AMC1300/Settings/FC7-11/: FMC Startup Failure
[MasterGM2000,ERROR] [frontend.cpp:903:frontend_init,ERROR] FC7 Initialization Failed
The value of code is: 0
[MasterGM2000,ERROR] [AMC1300.cpp:331:FC7_init,ERROR] /AMC1300/Settings/FC7-11/: FMC Startup Failure
[MasterGM2000,ERROR] [frontend.cpp:903:frontend_init,ERROR] FC7 Initialization Failed

Where code is the value:

  int code = 0;

  code += getRegister(limit, delay, fc7, "FMC_READY");
  code += getRegister(limit, delay, fc7, "FMC_ID_VALID");

  if (code != 2) {
    return 0;
  }
  int code = 0;

  code += getRegister(limit, delay, fc7, "FMC_READY");
  code += getRegister(limit, delay, fc7, "FMC_ID_VALID");

  if (code != 2) {
    return 0;
  }

So the FC7 is not liking the FMC I added. This could be because my bootleg setup.

15/03/2024 00:07

I noticed that the boost installation in /home/installation_testing/packages/boost does not properly link python with boost. However, /home/backup_installation_testing/packages/boost-1.53.0 does, so I just editted /home/installation_testing/packages/experiment/lxedaq/environment_setup/environment_variables.txt to look like:

[root@dhcp-10-163-105-238 environment_setup]# cat environment_variables.txt
GM2DAQ_DIR=/home/installation_testing/packages/experiment/lxedaq
CACTUS_ROOT=/home/installation_testing/packages/cactus
BOOST_ROOT=/home/backup_installation_testing/packages/boost-1.53.0
PUGIXML_ROOT=/home/installation_testing/packages/pugixml-1.8
ROOT_ROOT=/usr/include/root
MIDASSYS=/home/installation_testing/packages/midas
MIDAS_EXPTAB=/home/installation_testing/online/exptab
MIDAS_EXPT_NAME=DAQ
[root@dhcp-10-163-105-238 environment_setup]# cat environment_variables.txt
GM2DAQ_DIR=/home/installation_testing/packages/experiment/lxedaq
CACTUS_ROOT=/home/installation_testing/packages/cactus
BOOST_ROOT=/home/backup_installation_testing/packages/boost-1.53.0
PUGIXML_ROOT=/home/installation_testing/packages/pugixml-1.8
ROOT_ROOT=/usr/include/root
MIDASSYS=/home/installation_testing/packages/midas
MIDAS_EXPTAB=/home/installation_testing/online/exptab
MIDAS_EXPT_NAME=DAQ

This lets me use uHAL in python.

15/03/2024 00:12

Lawrence mentioned that the FC7 firmware needed to be updated. To do this, I follow the steps he gave in an email.

  1. cd /home/installation_testing/packages/experiment/lxedaq/environment_setup/environment_variables.txt
  2. source setup_environment.sh
  3. cd /home/installation_testing/packages/unifiedCCC/software/flash
  4. export CPLUS_INCLUDE_PATH="$BOOST_ROOT/include:$CPLUS_INCLUDE_PATH"
  5. Added the bottom line to the Makefile
LIBRARY_PATH = -Llib \
               -L/opt/cactus/lib \
               -L$(CACTUS_ROOT)/lib \
               -L$(CACTUS_ROOT)/uhal/uhal/lib \
               -L$(CACTUS_ROOT)/uhal/grammars/lib \
               -L$(CACTUS_ROOT)/uhal/log/lib \
               -L$(CACTUS_ROOT)/extern/pugixml/RPMBUILD/SOURCES/lib \
               -L$(CACTUS_ROOT)/extern/boost/RPMBUILD/SOURCES/lib \
               -L$(BOOST_ROOT)/lib
LIBRARY_PATH = -Llib \
               -L/opt/cactus/lib \
               -L$(CACTUS_ROOT)/lib \
               -L$(CACTUS_ROOT)/uhal/uhal/lib \
               -L$(CACTUS_ROOT)/uhal/grammars/lib \
               -L$(CACTUS_ROOT)/uhal/log/lib \
               -L$(CACTUS_ROOT)/extern/pugixml/RPMBUILD/SOURCES/lib \
               -L$(CACTUS_ROOT)/extern/boost/RPMBUILD/SOURCES/lib \
               -L$(BOOST_ROOT)/lib
  1. make
  2. cd bin
  3. ./programFC7 1 11 /home/installation_testing/packages/unifiedCCC/releases/fc7_unified_0x080107.mcs /home/installation_testing/packages/experiment/lxedaq/address_tables/FC7_CCC.xml (for help on what each parameter is, you can just do ./programFC7)
  4. Reboot the FC7, you can do this from telnet:
    1. telnet 192.168.1.41
    2. ? for list of command
    3. show_fru, note the fru ID in the left column for the FC7
    4. shutdown 15 wait a bit
    5. fru_start 15 wait a bit
  5. Check the FC7 is alive ping 192.168.1.11
    After doing all this the firmware should be properly updated, I was able to get past our errors:
  6. cd /home/installation_testing/packages/experiment/lxedaq/frontends/MasterGM2
  7. ./frontend -i 0 -e DAQ
    Output:
...
FC7_init(241): FC7 Ethernet Communication Check: 1/1
FC7_init(288): Slot 11: FC7 Firmware Hard Reset
FC7_init(298): Waiting 5 s ...
FC7_init(410): Slot 11: Write: Enabled Top SFP Ports
FC7_init(420): Waiting 15 s ...
FC7_init(449): Slot 11: Read: Enabled Top SFP Ports: 0
[MasterGM2000,ERROR] [AMC1300.cpp:450:FC7_init,ERROR] /AMC1300/Settings/FC7-11/: Enabled Top SFP Ports Failure
[MasterGM2000,ERROR] [frontend.cpp:903:frontend_init,ERROR] FC7 Initialization Failed
...
FC7_init(241): FC7 Ethernet Communication Check: 1/1
FC7_init(288): Slot 11: FC7 Firmware Hard Reset
FC7_init(298): Waiting 5 s ...
FC7_init(410): Slot 11: Write: Enabled Top SFP Ports
FC7_init(420): Waiting 15 s ...
FC7_init(449): Slot 11: Read: Enabled Top SFP Ports: 0
[MasterGM2000,ERROR] [AMC1300.cpp:450:FC7_init,ERROR] /AMC1300/Settings/FC7-11/: Enabled Top SFP Ports Failure
[MasterGM2000,ERROR] [frontend.cpp:903:frontend_init,ERROR] FC7 Initialization Failed

So now we move onto debugging SFP Port error

15/03/2024 00:50

bcb0e25ee79da2d8869e5e0666884c2b.png
I noitced the extra SFPs we have are actually loopback SFPs that are used just to test a port. We cannot plug anything into them. They look like the bottom SFP in the image above.

I think we have just enough SFPs for the setup, until we get a second crate where we should need 2 more.

15/03/2024 00:58

Now that the CENPA teststand is set back up, I can ssh into it with the old command, for example to see midas on localhost:8081 on my computer:

ssh -L 8081:localhost:8080 j.carlton@wombat.npl.washington.edu -p 22001
ssh -L 8081:localhost:8080 j.carlton@wombat.npl.washington.edu -p 22001